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(54) Digital manipulation of audio samples 

(57) The soundof a plurality of a selected instrument 
is derived from a single digitized audio sample of the 
selected instrument by storing the digitized audio sam- 
pie of the single selected instrument in a memory and 
manipulating copies of the digitized audio sample in par- 
allel in a plurality of digital processors corresponding in 
number to the plurality of the selected instrument. Each 



of the digital processors processes the digital audio 
sample in a slightly different time variant manner to pro- 
duce the effect of a plurality of instruments. The proc- 
essed digital audio samples are summed into a single 
digital sample which is converted to an analog signal 
which is sent to a speaker to produce the sound of the 
plurality of the selected instrument. 
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Description 

The present invention relates to the digital nnanipulation of audio samples, and in particular a method for manipulating 
a digitally sampled audio recording of a single instrument to produce the sound of a plurality of the same instrument. 
5 MIDI-controlled music synthesizers using waveform sampling technology are used extensively in the music and 

multimedia fields for their ability to create musical sounds that closely emulate the sound of acoustical music instruments. 
I^IDl is a music encoding process which conforms to the Music Instrument Digital Interface standard published by the 
International MIDI Association. MIDI data represents music events such as the occurrence of a specific musical note, 
e.g.. middle C, to be realized by a specific musical sound, e.g.. piano, horn, drum, etc. The analog audio is realized by 
10 a music synthesizer responding to this MIDI data. 

A major limitation of current MIDI music synthesizers is the lack of sufficient memory to store the entire sample of 
a wide range of an acoustic instrument's sounds. This inability to store many variations of. asound mean&thatthe music, 
synthesizer would need, for example, a separate sample for the sound of 1 violin, another sample for the sound of 4 
violins, yet another sample for the sound of 1 2 violins, and so on. Since each sample requires a great deal of memory, 
'5 most synthesizers on the market offer a limited selection of variations. 

It IS therefore an object of the invention to produce the sound of any number of a selected musical instrument from 
the sampled sound of a single one of the specified Instrument. 

This object is achieved by the invention claimed in claim 1 . 

An embodiment of the invention will now be described, by way of example, with reference to the accompanying 
^0 drawings, in which: 

FIG 1 shows a sampling audio synthesizer process embodying the present invention. 

FIG, 2 depicts the process of converting a recorded audo waveform to a digital sample. 

FIG. 3 depicts a more detailed diagram of the digital processing procedure. 

FIG. 4 depicts the digital samples generated by the digital processing procedure. 
^5 FIG. 5 illustrates a multimedia personal computer in which the present invention is embodied. 

FIG. 6 is a block diagram of an audio card in which the invention is embodied together with the personal computer 
in FIG. 5. 

FIG. 7 is a user interface to control the process of the present embodiment. 
FIG 8 is a block diagram of a music synthesizer in which the present invention is embodied. 
30 The sound of a single musical instrument differs from the sound of several musical instruments of the same type. 

To properly create these variations in conventional audio sampling synthesizers, separate audio samples are currently 
maintained within a music synthesizer thus increasing the memory storage requirements for each set of instruments. 
The present invention vastly reduces the storage problem by storing the audk> sample of a single instrument and ma- 
nipulating this audio sample data in specific ways to simulate the desired variation. 
55 FIG. 1 depicts the audb synthesizer process according to the principles of the present embodiment- This sampling 

audio synthesis process could be performed by a special purpose music synthesizer, alternatively, the process could 
be performed by a combination of software and/or hardware in a general purpose computer 

The audio sample contained in a sampling music synthesizer is a digital representatbn of the sound of an acoustic 
instrument. The audb sample may last for 5 to 10 seconds or more depending upon the musical instrument but only a 
•*o small portion of that sample is typically stored within the music synthesizer. The audio sample of a single musical in- 
strument, a violin, has been stored in the music synthesizer, but the sound of multiple instruments, twelve vblins. Is 
desired. This embodiment manipulates the audb sample of the one vblin to simulate the sound of twelve vblins by 
manipulating multiple copies of the single violin sample, e.g., by adding a different random, time variant value to the 
amplitude of each sample copy to simulate the time-based variation between multiple instrument performers. The plu- 
rality of manipulated audio samples are then summed to produce a single audio signal that emulates the sound of 
multiple instruments. This summed audio signal is converted to analog and amplified to produce the sound of twelve 
violins. This example can be extended to produce the sound of any number of vblins all from the original sound of a 
single violin or any of the other audb samples stored in memory. 

Groups of other instruments such as flutes may be created by other samples in memory; the actual sample used 
depends upon the instrument sound being synthesized. The random amplitude variation is introduced to simulate the 
natural variation between the selected instrument performers. 

The process begins by storing several musical samples in the audio sample memory in step 10. The audio sample 
memory is cither read only menrxDry (ROM) for a synthesizer whose sound capability is not changeable or a random 
access memory (RAM) for a synthesizer whose sound capability may be altered. The AUDIOVATION"* sound card 
manufactured by the IBM Corporation is of the altered type since the computer's hard disk memory stores the samples. 
Synthesizers such as the Proteus™* series by E-Mu Systems. Inc. have a set of samples in 4 to 16 MB of ROM and 
thus are of the fixed type. 

Next, the user or applicatbn selects one of the audb samples for further processing, step 11 . In the figure, the audio 
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sample selection input 1 3 is for a violin. The digital sample of the violin is passed to the digital processing step 1 5. yvhere 
the number of instruments input 17. in this case, the number of violins, twelve, and the degree of variation input 19 are 
received. The control is provided over the degree of variation between the simulated twelve violins to match the taste 
of the user, the style of music being played and so forth. 
5 In the digital processing step, the audio sample is copied to a plurality of processors, corresponding in number to 

the number of instruments desired. Each of these processors manipulates the sample in a slightly different time-variant 
manner The result of these manipulations is summed to form a digital audio sample of the desired number of Instruments. 
The digital processing step 15 is discussed below in greater detail with reference to FIG. 3. 

In step 21. the digital sound representation of the 12 vblins is converted to an analog audio signal. In step 23, the 
10 analog audio signal is amplified. Finally, in step 25, the actual sound of 1 2 violins is produced by an audio amplifier with 
speakers or by audio headphones. 

In the preferfed embodiment^ the audio^sample storage step 10, the audio^sample.selectioastepJI and thexiigital, 
processing step 15 are accomplished by computer software programs executed by a computer. The "computer" may 
be a stand alone general purpose computer equipped with a sound card or built-in sound software or it may be a computer 
IS chip within a specialized music synthesizer. The computer and audio card are discussed in greater detail with reference 
to FIGs. 5 and 6 below A sample user interface for a computer is shown in FIG. 7. The digital to analog conversion step 
21 is typically perfomied by a dedicated piece of hardware. A typical hardware component for such conversion is a 
codec, which produces an analog voltage corresponding to digital data values at a specific time interval. For example, 
44K digital audio data would be sent to a codec every 1/44,100 seconds and the codec's analog output would reflect 
20 each input digital data value. The codec is also used to convert analog audio entering the computer into a digital form. 
All synthesizers, including multimedia enabled computers, have Digital-to- Analog converters to produce the analog audio 
signal. For example, a suitable D to A converter is the Crystal Semiconductor Corp's codec chip CS4231. The analog 
audio amplification step 23 is performed by an analog amplifier. The actual production of sound, step 25. is accomplished 
by sending the amplified signal to audio speakers or audio headphones. Both the amplifier and the speakers or head- 
2S phones are normally separate pieces of hardware. They may be incorporated within the chassis of a music synthesizer 
or multimedia enabled computer, but they are usually distinct units. 

One of the primary advantages of this embodiment is limiting the number of audio samples in the audio sample 
memory, one of the nrK>st expensive part of a musical synthesizer. Another advantage of the present emfcxxiiment is that 
a more interesting sound is produced by the synthesizer. Typically, in a synthesizer, the last few samples in an audio 
30 sample are repeated over and over and are combined with an amplitude envelope to simulate the natural volume re- 
duction, i.e., decay, of an acoustic instrument's sound. The part of the sound where the repetition starts is thus the same 
sound repeated over and over at a reducing volume. This sound is very uniform since the same set of audio samples 
are used and has a very rK)n-musical feel. The sound of an actual acoustic instrument varies in small respects at all 
times and does not exhibit this repetitious characteristic. This embodiment modifies each audk) sample throughout the 
35 amplitude envelope in a digital processor in a time variant manner to provide a much more natural sound. 

The process of audb digital sampling is acconnplished where an audio waveform produced by a microphone, and 
possibly recorded on a storage medium, is sampled at specific time intervals. The magnitude of that sample at each 
point in time is saved digitally in memory. In a computer system, a sample is a binary representation of the amplitude 
of an analog audio signal measured at a given point in time; a sample really is just an amplitude measurement. By 
^ repeated measurements of the analog signal at a sufficiently high frequency, the series of binary representations can 
be stored in memory and be used to faithfully reproduce the original analog signal by creating an analog voltage that 
follows the stored values in memory over the time intervals. 

The magnitude of th6 audio dat^ reffects the loudness of thb audib signal; a I5uder sound produces a larger ddta 
magnitude. The rate at which the audio data changes reflects the frequency content of the audb signal; a higher fre- 
quency sound produces a larger change in data magnitude from data sample to data sample. 

In FIG. 2, the set of vtoltn samples 43 Is stored in memory as a series of 16-bit data values This storage could be 
different lengths, e.g., 8-bit, 12-brt, depending upon the desired quality of the audio signal. The box 45 to the right 
illustrates an example of data stored in the first 8 violin samples. The analog audio signal is later formed by creating an 
analog voltage level corresponding to the data values stored In box 45 at the sampling interval. The graph at the bottom 
shows the resultant analog waveform 40 created from these first 8 violin audio samples after the digital-to-analog con- 
version where data point 41 of this graph is an example of the data of box 45 at sample time #2. 

The binary representation of the analog signal is measured in number of bits per sample; the more bits, the more 
accurate representation of the analog signal. For example, an 8-bit sample width divides the anatog signal measurement 
into 28 units meaning that the analog signal is approximated by 1 of a maximum of 256 units of measurements. A 8-bit 
sample width introduces noticeable errors and noise into the samples. A 16-bit sample width divides the analog signal 
measurement into 2^^ units and so the error Is less than 1 part in 64K, a much more accurate representation. 

The number of samples per second determines the frequency content, the more samples, the increased frequency 
content. The upper frequency limit is approximately 1/2 the sampling rate. Thus, 44K samples per second produce an 
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upper frequency limit of about 20 kHz, the limit of hurnan hearing. A sample rate of 22K samples per second produce 
a 10 kHz upper limit, and high frequencies are lost and the sound appears muffled. The resultant audio signal, given 
the limits of sample width and sample rate, can thus follow the more intricate movements of an analog signal and re- 
produce the sound of the sampled musical instrument with extreme accuracy. However, extreme accuracy requires 
5 substantial data storage, one 4-second violin sample recorded at 16-bits and 44K samples per second requires (4 
seconds)x(2 audio channels for stereo)x(2 bytes for 16-bits)x(44.100 for 44K samples per second) or about 700 KB. Up 
to 5 or more violin samples may be needed to cover the entire pitch range of a violin meaning that 3500 KB are required 
just for one musical Instrument. The samples for 4 vbllns would be another 3500 KB as would the samples for 1 2 violins. 
To cover all of the varlatbns for all of the instruments of the orchestra represents a sizable amount of storage. Thus. 
10 the reader can appreciate the storage problems of the current audio synthesizer. 

The present embodiment requires only the storage of a single violin. As discussed above, and in greater detail 
below, to obtaia the sound ol multiple violins, the digitalprocessing mtcromanipulates the $iogle.violin sannplato emulate . ... 
the multiple violin sound. 

Another advantage of this embodiment is that the sound of an exact number of instruments may be produced. 
IS Modern synthesizers may offer samples of 1 violin and of 30 violins, but not of intermediate numbers of violins due to 
the previously mentbned memory limitations. With this embodiment, the user may select the sound of any specific 
number of Instruments, 10 violins for example, and the synthesizer will produce the appropriate sound. Small variations 
are introduced into the samples providing variation In the resultant sound. Sampling technology suffers from producing 
the exact same sound each time the sample is played back. The sound may be an accurate representation of the musical 
instrument, but the sound can become less Interesting due to the lack of variatbn each time it is played back. 

If the user wanted the sound of a single instrument, the digital processing could be effectively bypassed. Nonethe- 
less, as an added advantage of the embodiment, the user may still want to digitally process the signal to introduce small 
variations and make the signal more Interesting than prior art sampling technologies. 

By ■micromanipulating" the audio samples, the Inventors Intend to add small variations between the original audio 
sample and between the manipulated audo samples produced by the digital processors. The micromanipulations have 
to be sufficient to create a perceptible difference between the sample sets produced by two different processors. On 
the other hand, the micromanipulations must not be so great as to render the manipulated sample unrecognizable as 
the originally sampled instrument. The idea behind the embodiment is to produce the sound of many of the same in- 
strument, not to produce the sound of many new and different instruments. 
30 As mentioned above, a random number generator may be used In oonjunctran with this embodiment. Preferably, 

the random number is used as a seed for the digital processor, unless the degree of variatbn Is small, entirely random 
processing for each sample would tend to create nonmusical sounds. From the random seed, the processor would 
determine the conditions to start the micromanipulation; the subsequent audk> samples, the adjustments to the gain and 
so forth would flow from the Initial starting condltbns within the envelope chosen. 
3S FIG. 3 shows greater detail of the digital processing procedure. The number of processes or tone generators 50-53 

are set up or called according to the number of instruments chosen by the user or application. From a set of violin 
samples 54, a corresponding number of Individual violin samples 55-58 are fed to the processes 50-53, and individually 
processed in parallel. The resulting manipulated digital samples 60-63 are then summed digitally 64 to form the com- 
posite digital sample 65 for the sound of multiple violins at that point in time. 

Time variations are introduced to simulate minor amplitude or pitch changes of specify simulated violins. The time 
variations may be Influenced by a random number generator either as a seed or to introduce small random variatbns 
within a pemiltted envetope. The envelope dimensions are based on the input degree of variation. The digital processing 
has components that determine the specific values of gain, tone, and time variation. This process is repeated at suc- 
cessive times to form the composite sound of the multiple violins over time. 

In FIG- 3, the user has Input the requirement to create the sound of 4 violins from the sample of a lone violin. 
Processes #1-4 may manipulate each of the four samples using time variant Gain and Filter functions. The input or the 
degree of variation variable controls the data range over which these functkxis may vary. 

As shown in the equations below, each process may modify the sample's gain, that Is, Its amplitude and tone by 
digital filtering. 
so At time =t1 

Vsum, = Sample, G,(t,)F,(t,)+Sample,,G2(t,)F2(t,) 

+Sample,eG3(t,)F3(ti)+Sample22(G4(t,)F4(t, ) 

where Vsum, is the sum of the manipulated signals; Samplei , Sample^ , SampleiQ and Sampleg^ are amplitudes 
from the set of audio samples at particular instants in time; (G, (t^ ). Ggit, ), Ggpi ) and G4(ti ) are time variant gain functions 
for each of the processors, at time t,; and Fi(t,).F2(ti). FaCt,) and F4(tO are time variant filter functions at time t^. 
The gain functions at time=ti might be Gi=1.00. G2=0.95. 63=1.11. 64=093 within the respective processes, thus 
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emphasizing Sample #18 since gain is greater than 1.0 and deemphasizing samples #11 and #22 since gain is less 
than 1 0. The gains at time=t2 might be 6^=1 .02. G2=0.92, 63=1 .03. 64=0.99 which is similar to t=t1 but shows a slow 
variance. This micromanipulation would continue such that the samples that are emphasized and deemphasized vary 
over time as happens when 4 violin players play concurrently. Similar variations would occur in the filtering functions 

5 with time. The end result would be to vary the upper frequency content, and the pitch of the four instruments to simulate 
the minor tone variations produced by 4 violin players playing concurrently. Other processes could certainly be Included 
to produce variation in the treatment of the samples. Time variations would be included to simulate the fact that 4 violin 
players never play exactly concurrently. It is important to note that the micromainpulations are time variant with respect 
to each other, so that the processes do not travel through time In lock step with each other. Although less preferred, one 

10 of the processes could be no change at all to the initial audio sample. 

The degree of the variance is influenced by the user, but the distribution of this variance is controlled by the digital 
processing process.. One example is ,tQ distribute the variance as a statistical "beill curve al>out.the norm, thus simulating 
the fact that most musicians play near the nominal condition while fewer and fewer musicians proportionally play at 
conditions approaching the outer limits of the distribution. The amount of variation between the individual simulated 

15 musical instruments is governed by the nature of the instruments and the taste of the user. The sound of multiple strings 
for example would allow more variation, i.e. a wider bell cun^e. than the sound of multiple clarinets, since the clarinet 
sound has a more distinct quality and would more easily appear "out of tune". In the preferred embodiment, the variations 
could adhere to a "bell" cun^e distribution, although other distributions are also appropriate, where the 3-sigma statistical 
variation is approximately 15% for amplitude. 30 cents (1 musical half-step is 100 cents) for pitch, and 30 milliseconds 

20 in time. 

FIG. 4 illustrates the manipulation of the audio waveform represented by the samples of 1 violin when converted 
into the audio waveform represented 4 violins. The original audio waveform 70 of 1 violin is represented by the samples 
stored in memory. To generate the sound of 4 violins, 4 processes 71-74 are started in the digital processing procedure. 
Each process modifies the digital data representing the single violin sound as shown by the 4 'modified' audio waveforms 

25 76-78. The audio waveforms shown represent the Individual sounds of the 4 simulated 'individual' violins. The digital 
data for the 4 modified audio waveforms digitally Is then summed 79 to produce the digital data for a "group' of 4 violins, 
as represented by the audio waveform 80 for 4 violins. 

As mentioned previously, the Invention may be embodied In a general purpose computer equipped with a sound 
card or sound circuitry and appropriate software, or on a special purpose audio synthesizer For example, computers 

30 in the IBM RS/eOOO^"^ or PowerPC^** series of computers equipped with an advanced sound card could be used. 

In FIG. 5. a computer 100. comprising a system unit 111. a keyboard 112. a mouse 113 and a display 114 are 
depicted. The system unit 1 1 1 includes a system bus or plurality of system buses 1 21 to which various components are 
coupled and by which communication between the various components is accomplished. The microprocessor 122 Is 
connected to the system bus 121 and Is supported by read only memory (ROM) 123 and random access memory (RAM) 

35 1 24 also connected to system bus 1 21 . A microprocessor in the I BM multimedia PS/2 series of computers is one of the 
Intel family of microprocessors Including the 386, 486 or Pentium™ microprocessors. However other microprocessors 
included, but not limited to, Motorola's family of microprocessors such as the 68000. 68020 or the 68030 microprocessors 
and various Reduced Instruction Set Computer (RISC) microprocessors such as the PowerPC or Power 2 chipset man- 
ufactured by IBM, or other processors by Hewlett Packard, Sun. Intel. Motorola and others may be used in the specific 

40 computer. 

The ROM 123 contains among other code the Basic Input-Output system (BIOS) which controls basic hardware 
operations such as the interaction and the disk drives and the k6ytx>ard. The RAM 124 is the main memory Into which 
the operatiiig system ancTapplication programs are Ibad^jd: JYi§ mernory management chip 1251s connected' to XM 
system bus 121 and controls direct memory access operations including, passing data t}etween the RAM 24 and hard 
disk drive 126 and floppy disk drive 127. The CD ROM 132 also coupled to the system bus 121 is used to store a large 
amount of data, e.g., a multimedia program or presentation. 

Also connected to this system bus 1 21 are various I/O controllers: The keyboard controller 1 28, the mouse controller 
129, the video controller 130, and the audio controller 131. As might be expected, the keyboard controller 128 provides 
the hardware interface for the keyboard 112, the mouse controller 129 provides the hardware interface for mouse 113. 
50 the video controller 130 is the hardware interface for the display 114, and a printer controller 131 Is used to control a 
printer 1 32. The audio controller 1 33 is the amplifier and hardware interface for the speakers 1 35 which the processed 
audio signal to the user. An I/O controller 140 such as a Token Ring Adapter enables communication over a network 
1 46 to other similarly configured data processing systems. 

An audio card which embodies the present inventbn, is discussed below in connection with FIG. 6. Those skilled 
In the art would recognize that the described audio card is merely illustrative. 

The audio control card 1 33 is an audio subsystem that provides basic audio function to computers made by the IBM 
Corporation and other compatible personal computers. Among other functions, subsystem gives the user the capability 
to record and play back audio signals. The adapter card can be divided into two main sections: DSP Subsystem 202 



55 



OOCID: <EP 0690434A2_I_> 



5 



V 



IS 



2S 



30 



35 



4S 



SO 



SS 



EP 0 690 434 A2 

and Analog Subsystem 204. The DSP Subsystem 202 makes up the digital section 208 of the card 200. The rest of the 
components make up the analog section 210. Mounted on the adapter card 200 is a digital signal processor (DSP) 212 
and an analog coding/decoding (CODEC) chip 21 3 that converts signals between the digital and analog domains. 
The DSP Subsystem portion 202 of the card handles all communications with the host computer All bus interfacing 
5 is handled within the DSP 212 itself. Storage can be accomnrKxlated in local RAM 214 or local ROM 215 the DSP 212 
uses two oscillators 216, 218 as its clock sources. The DSP 212 also needs a set of external buffers 220 to provide 
enough current to drive the host computer bus. The bi-directional buffers 220 redrive the sigials used to communicate 
with the host computer bus. The DSP 202 controls the CODEC 21 3 via a serial communications link 224. This link 224 
consists of four lines: Serial Data. Serial Clock. CODEC Ckxk and Frame Synchronization Clock. These are the digital 
10 signals that enter the analog section 204 of the card. 

The analog subsystem 204 is made up of the CODEC 214 and a pre-amplifier 226. The CODEC 213 handles all 
the AnalogtocOioiial (A/R)^odQjgitaN(>^Anatag;(D/A) conversions by communicating with the DSP 21 2.tQ transfer data . 
to and from the host computer. The DSP 212 may transform the data before passing It on to the host. Analog signals 
come from the outside world through the Line Input 228 and Microphone Input 230 jacks. The signals are fed into the 
pre-amplifier 226 built around a single operational amplifier. The amplifier 226 conditions the input signal levels before 
they connect to the CODEC 213. In the future many of the components shown in the audio card may be placed on the 
motherboard of a multimedia enabled computer. The process may be perfornied by the computer and audio card depicted 
in FIGs. 5 and 6 respectively in a several different implemeniatbns. The storage of the audio samples and the micro- 
manipulation processing may be accomplished by a software implementation in the main computer. Audio samples 154 
and digital processing program 156 are stored in permanent storage on the hard disk 126 or a removable floppy disk 
placed in the floppy drive 1 27 and read into RAM 1 24. The processor 1 23 executes the instructions of the digital process- 
ing program to produce a new digitized sample for the plurality of instruments. The sample is sent to the audk> card 1 33 
where the signal Is converted to analog signals which are in tum sent to the amplifier and speakers 1 35 to produce the 
actual sound which reaches the user's ears. The user may interact with the digital processing program 156 directly 
through the use of a graphical user interface to select the instrument, the degree of variance and the desired number 
of instruments. Alternatively, the user may interact with the user interface of an audio program 158 which makes the 
actual call to the digital processing program 156 with the required parameters. 

In the alternative, the actual digital processing may be accomplished by the DSP 212 on the audio card 133. In this 
embodiment, the digital processing program would toaded into the DSP 212 or local RAM 214 from permanent storage 
at the computer. The audio samples may be stored in permanent storage at the computer or In local ROM 215, The 
digital processing would be accomplished by the DSP 212 which would send the digital sample to the CODEC 213 for 
processing to an analog signal. It woukJ be likely that a portion of the digital processing program 156 would still be 
required at the computer to provide a graphbal user interface or an interface to audio applications which request the 
digital processing services. 

Those skilled in the art would recognize that other embodiments within a general purpose computer are possible. ' 
Referring to FIG. 7, a graphical user interface (GUI) 290 is described as foltows. The GUI action bar 295 is divided 
into three subsections: File I/O 300, Audio Information (330), and MIDI Informatbn 303. respectively When the File I/O 
option 300 is selected, an area 305 is devoted to displaying waveform data is shown. Various options on the pull-down 
would display different waveforms. For example, the input waveform data 310. that Is. the original unmodified audio 
data, is shown when the input option 311 on the pull down is selected The input waveform graph 310 represents this 
waveform data as a ptotorlal view of the spectrum plot. After alteration of the data occurs, a pictorial view of the spectrum 
plot Is available in output data graph 320. a selectkxi of the output option in the menu pulldown. This audio data represents 
the micromanipulated sample data. The file I/O menu pull down could also include a select instrument option. 

The user may request modification of the audio sample by selecting the audb 301 and MIDI 303 section. Audio 
information is selected via a control box 330 which contains several controls 331-333. e.g., dials, set to some values. 
The dials may control a degree of variation value, a variable sampling rate (Fs), and a scaling factor for the envelope's 
amplitude for example. The selection of the MIDI optton 303 causes MIDI controls 340. 350 to popup which contain yet 
other controls for, values for volume. MIDI ports, and Instrument Selection (timbre). As the user experiments while 
controlling Audio and MIDI control boxes 340, 350. the pictorial view of the audk> waveform data 320 dynamically changes 
relative to the original audio input samples 310. One skilled in the art would recognize that many other GUIs could be 
used to control the process. For example, a simple dialogue box which contains entry fields for the instrument type, 
number of instruments and degree of variation might be used. 

In FIG. 8. a special audio synthesizer 400 which is emulating an ensemble of instruments is depicted. MIDI data 
enters the synthesizer at Its MIDI-IN connector 401 and is decoded by its MIDI decode circuitry 402. The MIDI data 
consists primarily of MIDI controls 402 and MIDI note data 403. From the MIDI control data 402, the MIDI control btock 
404 selects a sampled wavefomi from memory 405 for each of the synthesizer's voice bkx;ks 406. In the example shown, 
the voice #1 bkx:k obtains a violin sample and the voice #2 bkxsk obtains a flute sample and so forth 

For the sake of simplicity, only the violin sample processing is depicted Similar components exist for each of the 
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other voices. From the MIDI note data 403, the MIDI note data block 407 determines the fundamental frequency of the 
note from the MIDI note command's key number and the volume of the note from the MIDI note command's velocity. 
This data is combined with the sample waveform from the voice block 406 modified by the Modify Wavefonn block 408. 
The result 409 in this example is a sample of a violin whose frequency and volume are determined by the MIDI note 

s data and whose start and stop times are determined by the timing of the corresponding MIDI Note-ON command and 
Note-Off command. The modified violin sample 409 is then modified by the Micro Waveform Control block 410 which 
generates the sound of multiple violins by the Digital Processing procedure as discussed above with reference to FIG. 
3. The resultant set of audio samples Is converted Into separate stereo left and right channel samples by the Create 
Stereo Sample bkx:k 412 under control of the MIDI Control 411 . 

10 The other voices from the Waveform Vobe Block 406 are treated in a manner similar to Voice #1: the violin, as 

described above. The stereo sanriples from all of these voices are combined by the stereo audio mixer 413 into one set 
of stereo audio jMKPplas416»Tl^ae^mplesafeccrw into a stereo analog signaUISUyJhe.CodecdigitajriOranalog 
circuitry 414 and this analog signal is sent to an external audio amplifier and speakers (not illustrated) to be converted 
into sound. 

IS The following pseudo code illustrates one possible embodiment of a portion of the algorithmic technique for a MIDI 

manipulation according to the present inventbn: 



20 



2S 



3k, 



Ai(n) » Arl(n); 

where (Ai(n) is the time-varying amplitude controller for the i the 
sample value, and r is some random factor; Fs is the sampling frequency. 
(Note: the digital storage of a waveform's amplitude range is aseumed to 
be +/-32767 unite where a table containing waveform aMplitudee are 
assumed* ) 



MainO 
{ 

for (n»0;n<( Samples ) jn-t- +){ 

/*input number of samples for each instriunent and amplitudes */ 
I{n} ss Bcanf (aample^time{n} ) ; 
30 old_audio_amp{n} scanf (sample^amplitudefn} ; 

/* calculate amplitude threshold for each instrument by using */ 
/* a factor associated with instrument 
ampXitude_thre8hold{n} t ""I {n}/( rand (factor*! .0) , 
calculate randomized values */ 

Call Random_amp(old_audio_ainp{n} , amplitude_thre8hold{n} , 
35 randon_value8{n} ; 

inst r_new_ampl itudes { n } : =r andom^values {n > ; 
/♦output MIDI amplitude data */ 
output_port ( instr_new_amplitudes{n} ) ; 

>^ 

40 } /* end of main*/ 

Procedure Random_amp(old_audio_amp{n}, amplitude threaholdin} , 
^ randoro_valuea{nlI^ ~ 

/^compute new, randomized, Ai samples */ 
45 for (n"0;n<nSample8;n'f -t-) { 

randoro_value8{n} 

amplitude_thre8hold{n}*(old audio amp{in}-Mn8tr last amplitudeeinV * 
inBtr_la8t_amplitude8{n}} ; " ~ - ^ ' 

/* save amplitude values for next iteration of samples */ 
in8tr_la8t_amplitudeB{n} :» random_values{n}; 

so ^ 

}/* end of Random^ampO */ 



Claims 



1. A method for producing the sound of a plurality of a selected instrument from a single digitized audio sample of the 
selected instrument, comprising the steps of: 



DOCIO: <EP__069(M34A2J_> 



7 



EP 0 690 434 A2 



storing the digitized audio sample of the single selected instrument in a memory; 

manipulating copies of the digitized audio sample in parallel in a plurality of digital processors corresponding 
in number to the plurality of the selected instrument, each digital processor processing the digital audio sample in 
a slightly different time variant manner; 

summing the processed digital audio samples; and 

converting the summed digital audio sample to an analog signal sent to a speaker to produce the sound of 
the plurality of the selected instrument. 

2. The method claimed in claim 1 further including the step of calling the plurality of digital processors in response to 
a selection of the number of the plurality of the selected instrument. 

3.. ^Thf^ method ciaimedJacClwn-A iur^e^compasmg the step of altering the processing by ihe.pJifiality.of digital proc- 
essors in response to a degree of variation parameter. 

4. The method claimed in claim 1 wherein the nnantpulating in each digital processor is at least in part performed 
according to a random number generator. 

5. A system for producing the sound of a plurality of a selected instrument from a single digitized audio sample of the 
selected instrument, comprising: 

a memory for storing the digitized audio sannple of the single selected instrument; 

a plurality of digital processors for manipulating copies of the digitized audio sanrple in parallel, the plurality 
of digital processors corresponding in number to the plurality of the selected instrument, each digital processor 
processing the digital audio sample in a slightly different time variant nnanner; 

means for summing the processed digital audio samples: £ind 

a digital to analog convertor for converting the summed digital audio sample to an analog signal sent to a 
speaker to produce the sound of the plurality of the selected instrument. 

6. The system claimed in claim 5 further comprising a means for calling the plurality of digital processors in response 
to a selection of the number of the plurality of the selected instrument. 

7. The system claimed in claim 5 further comprising means for altering the processing by the plurality of digital proc- 
essors in response to a degree of variation parameter. 

8. The system claimed in claim 5 further comprising a random number generator wherein the processing in each digital 
processor Is at least in part performed according to the random number generator. 

9. The system claimed in claim 5 further comprising: 

a system bus coupled to the memory for passing data and instructions between components in the system; 
a display coupled to the system bus for presenting a user interface for control of the system, wherein a user 
makes inputs for the number of the plurality of the selected instrument and the degree of variatbn parameter 

10. The system claimed In claim 5 further comprising: 

an audio card on which the digital to analog convertor Is placed. 

11. The system claimed in claim 7 wherein an envelope in which the manipulating is bound is selected according to the 
degree of variation parameter 

12. The system claimed in claim 11 wherein the envelope is also selected according to the selected instrument. 
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